The influence of semantics in IR using LSI and K-means clustering techniques

In this paper we study the influence of semantics in the information retrieval preprocessing. We concretely compare the reached performance with stemming and semantic lemmatization as preprocessing. Three techniques are used in the study: the direct use of a weighted matrix, the SVD technique in the LSI model and the bisecting spherical k-means clustering technique. Although the results seem not to be very promising, we believe that they should be improved in the future. 1. BACKGROUND AND MOTIVATION The Information Retrieval (IR) models used in this work are classified within the vector space model, included in the classic model. The actual models used are the generalized vector space model and the Latent Semantic Indexing (LSI) [1]. These models are based in the well-known matrix of terms by documents, which generally is a weighted matrix and rarely a frequency matrix [2]. The terms by documents matrix is constructed from a collection of documents. The process to obtain this matrix requires a preprocessing of that collection. There are various techniques in the preprocessing part, each one handle one or more aspects, i.e. reduce the number of terms that represent the collection, treatment of related words, etc. After the preprocessing of the collection, a frequency matrix is constructed, which usually is transformed to a weighted matrix. There are many schemes to weight a frequency matrix, but a reasonable election is to use “term frequency” as the term frequency component and to use “inverse document frequency” as the collection frequency component [14], this scheme is used in this work. With the weighted matrix we model the information retrieval system induced by the document collection. But we assume, too, two others models, a LSI model (using the SVD technique) and a clustering model (using the bisecting spherical k-means algorithm [9]). The criteria used to evaluate the experiments, has been the average precision-recall ratio [1]: